{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MXNet Basics - Linear Regression using MXNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MXNet uses data in the form of Data **Iterators**. The code below illustrates how to encode a dataset into an iterator that MXNet can use. The data used in the example is made up of 2d data points with corresponding integer labels. The function we are trying to learn is:\n",
    "\n",
    " y = x<sub>1</sub>  +  2x<sub>2</sub> ,\n",
    " \n",
    " where (x<sub>1</sub>,x<sub>2</sub>) is one training data point and y is the corresponding label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Training data\n",
    "train_data = np.array([[1,2],[3,4],[5,6],[3,2],[7,1],[6,9]])\n",
    "train_label = np.array([5,11,17,7,9,24])\n",
    "batch_size = 1\n",
    "\n",
    "#Evaluation Data\n",
    "eval_data = np.array([[7,2],[6,10],[12,2]])\n",
    "eval_label = np.array([11,26,16])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have the data ready, we need to put it into an iterator and specify parameters such as the 'batch_size', and 'shuffle' which will determine the size of data the iterator feeds during each pass, and whether or not the data will be shuffled respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_iter = mx.io.NDArrayIter(train_data,train_label, batch_size, shuffle=True,label_name='lin_reg_label')\n",
    "eval_iter = mx.io.NDArrayIter(eval_data, eval_label, batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, we have made use of NDArrayIter, which is used to iterate over numpy arrays. In general, there are many different types of iterators in MXNet based on the type of data you will be using. Their complete documentation can be found at:  http://mxnet.io/api/python/io.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MXNet Classes\n",
    "\n",
    "1. Model Class: The model class in MXNet is used to define the overall entity of the model. It contains the variable we want to minimize, the training data and labels, and some additional parameters such as the learning rate and optimization algorithm are defined at the model level.\n",
    "\n",
    "2. Symbols: The actual MXNet network is defined using symbols. MXNet has different types of symbols, including data placeholders, neural network layers, and loss function symbols based on our requirement.\n",
    "\n",
    "3. IO: The IO class as we already saw works on the data, and carries out operations like breaking the data into batches and shuffling it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MXNet uses **Symbols** for defining a model. Symbols are the building blocks of the model and compose various components of the model. Some of the parts symbols are used to define are:\n",
    "1. Variables: A variable is a placeholder for future data. This symbol is used to define a spot which will be filled with training data/labels in the future when we are trying to train the model.\n",
    "2. Neural Network Layers: The layers of a network or any other type of model are also defined by Symbols. Such a *symbol* takes one of the previous symbols as its input, does some transformation on them, and creates an output. One such example is the \"Fully Connected\" symbol which specifies a fully connected layer of a network. \n",
    "3. Output Symbols: Output symbols are MXNet's way of defining a loss. They are suffixed with the work \"Output\" (eg. the SoftmaxOutput layer\" . You can also create your [own loss][https://github.com/dmlc/mxnet/blob/5b6a0eeee174f28ff0272d17748513ecd52a9ebe/docs/tutorials/r/CustomLossFunction.md#how-to-use-your-own-loss-function] Some examples of existing losses are: LinearRegressionOutput, which computes the l2-loss between it's input symbol and the actual labels provided to it, SoftmaxOutput, which computs the categorical cross-entropy. \n",
    "\n",
    "The ones described above, and other symbols are chained one after the other, servng as input to one another to create the network topology. More information about the different types of symbols can be found [here][http://mxnet.io/api/python/symbol.html]\n",
    "    \n",
    "    \n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = mx.sym.Variable('data')\n",
    "Y = mx.symbol.Variable('lin_reg_label')\n",
    "fully_connected_layer  = mx.sym.FullyConnected(data=X, name='fc1', num_hidden = 1)\n",
    "lro = mx.sym.LinearRegressionOutput(data=fully_connected_layer, label=Y, name=\"lro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above network uses the following layers:\n",
    "\n",
    "1. FullyConnected: The fully connected symbol represents a fully connected layer of a neural network (without any activation being applied), which in essence, is just a linear regression on the input attributes. It takes the following parameters:\n",
    "            a. data: Input to the layer (specify the symbol whose output should be fed here)\n",
    "            b. num_hidden: Number of hidden dimension which specifies the size of the output of the layer\n",
    "    \n",
    "    \n",
    "2. Linear Regression Output: Output layers in MXNet aim at implementing a loss. In our example, the Linear Regression Output layer is used which specifies that an l2 loss needs to be applied against it's input and the actual labels provided to this layer. The parameters to this layer are:\n",
    "            a. data: Input to this layer (specify the symbol whose output should be fed here)\n",
    "            b. Label: The training label against whom we will compare the input to the layer for calculation of l2 loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note - *Naming Convention*: the label variable's name should be the same as the label_name parameter passed to your training data iterator. The default value of this is 'softmax_label', but we have updated it to lin_reg_label in this tutorial as you can see in Y = mx.symbol.Variable('lin_reg_label') and train_iter = mx.io.NDArrayIter(..., label_name=\"lin_reg_label\")**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the network is stored into a *Module*, where you define the symbol who's value is to be minimised (in our case, lro or the lin_reg_output\"), the learning rate to be used while optimization and the number of epochs we want to train our model on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = mx.mod.Module(\n",
    "    symbol = lro ,\n",
    "    data_names=['data'], \n",
    "    label_names = ['lin_reg_label']# network structure    \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the network we have created in order to visualize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"254pt\"\n",
       " viewBox=\"0.00 0.00 214.00 254.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 250)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-250 210,-250 210,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-24.8\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- fc1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>fc1</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-152 -7.10543e-15,-152 -7.10543e-15,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-125.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n",
       "</g>\n",
       "<!-- fc1&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>fc1&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-83.7443C47,-75.2043 47,-66.2977 47,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-93.8971 42.5001,-83.897 47,-88.8971 47.0001,-83.8971 47.0001,-83.8971 47.0001,-83.8971 47,-88.8971 51.5001,-83.8971 47,-93.8971 47,-93.8971\"/>\n",
       "</g>\n",
       "<!-- lin_reg_label -->\n",
       "<g id=\"node3\" class=\"node\"><title>lin_reg_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-123\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-118.8\" font-family=\"Times,serif\" font-size=\"14.00\">lin_reg_label</text>\n",
       "</g>\n",
       "<!-- lro -->\n",
       "<g id=\"node4\" class=\"node\"><title>lro</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"150,-246 56,-246 56,-188 150,-188 150,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"103\" y=\"-212.8\" font-family=\"Times,serif\" font-size=\"14.00\">LinearRegressionOutput</text>\n",
       "</g>\n",
       "<!-- lro&#45;&gt;fc1 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>lro&#45;&gt;fc1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M80.5686,-179.148C75.113,-170.186 69.3635,-160.74 64.1943,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"85.8939,-187.897 76.8505,-181.695 83.2941,-183.626 80.6944,-179.355 80.6944,-179.355 80.6944,-179.355 83.2941,-183.626 84.5383,-177.015 85.8939,-187.897 85.8939,-187.897\"/>\n",
       "</g>\n",
       "<!-- lro&#45;&gt;lin_reg_label -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>lro&#45;&gt;lin_reg_label</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.377,-179.237C131.257,-169.578 137.481,-159.353 142.955,-150.36\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"120.106,-187.897 121.462,-177.015 122.706,-183.626 125.306,-179.355 125.306,-179.355 125.306,-179.355 122.706,-183.626 129.15,-181.695 120.106,-187.897 120.106,-187.897\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x10df64250>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mx.viz.plot_network(symbol=lro)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have defined the model structure, the next step is to train the parameters of the model to fit the training data. This is done by using the **fit()** function of the **Module** class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.fit(train_iter, eval_iter,\n",
    "            optimizer_params={'learning_rate':0.01, 'momentum': 0.9},\n",
    "            num_epoch=1000,\n",
    "            batch_end_callback = mx.callback.Speedometer(batch_size, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Using a trained model: (Testing and Inference) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Once we have a trained model, we can do multiple things on it. We can use it for inference, we can evaluate the trained model on test data. This is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 11.],\n",
       "       [ 26.],\n",
       "       [ 16.]], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Inference\n",
    "model.predict(eval_iter).asnumpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also evaluate our model for some metric. In this example, we are evaulating our model's mean squared error on the evaluation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mse', 0.0)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Evaluation\n",
    "metric = mx.metric.MSE()\n",
    "model.score(eval_iter, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let us try to add some noise to the evaluation data and see how the MSE changes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mse', 0.010000076144933701)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Evaluation Data\n",
    "eval_data = np.array([[7,2],[6,10],[12,2]])\n",
    "eval_label = np.array([11.1,26.1,16.1]) #Adding 0.1 to each of the values \n",
    "eval_iter = mx.io.NDArrayIter(eval_data, eval_label, batch_size, shuffle=False)\n",
    "\n",
    "model.score(eval_iter, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can create your own metrics and use it to evauate your model. More information on metrics here: http://mxnet-test.readthedocs.io/en/latest/api/metric.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
